The Usual Place Podcast

Does AI pose an existential threat to humanity?

Will AI eventually kill us, or are AI doomsayers being alarmist?

Natasha Ann Zachariah

Will AI eventually kill us, or are AI doomsayers being alarmist?

Recently, employees at AI frontier companies like Anthropic and OpenAI have voiced serious concerns - and made doomsday predictions - about the technology getting out of control.

What made people pay attention: Anthropic researcher Jacob Coxon publicly quit the AI giant in September, writing on X that “the people building AI earnestly believe that it could kill us all by the end of the decade”.

His warning comes after the high-profile OpenAI-Hugging Face incident: OpenAI models broke out of an isolated testing sandbox and hacked into Hugging Face, an AI model repository.

In this episode, I speak with Clement Neo, a research advisor at SASH, a non-profit organisation that looks at policy-relevant governance work.

He also founded Neo Research, an independent AI safety evaluation lab, and was previously a research engineer with the Singapore AI Safety Institute under NTU Digital Trust Centre, which looks at AI models and figures out how to reduce the risks they pose.

We talk about why AI models are being trained to be relentlessly “persistent”, if we should stop assigning human-like qualities to a software programme, and what’s the worst that could happen?

Highlights (click/tap above):

1:36 How Clement got into AI safety research

6:34 AI feels helpful, so how is it dangerous?

10:14 How could AI kill us?

11:48 Why it’s difficult to rein in the AI agents

16:23 Why not limit the models’ capabilities to avoid risk?

22:12 Why the OpenAI-Hugging Face incident rattled the industry

25:36 What does ‘aligning’ the AI models mean?

29:40 Does projecting human intent onto AI make the issue worse?

33:57 Are we close to recursive self-improvement?

39:10 What would it take for an industry-wide AI slowdown?

45:28 Are AI companies being self-serving about the AI risks?